Method and apparatus for providing fast reroute of a multicast packet within a network element to an available port associated with a multi-link trunk

ABSTRACT

A method, apparatus and computer program product for providing a fast re-route of a multicast packet within a network element to an available port associated with a multi-link trunk is presented. A packet is received by a Forwarding Data Unit (FDU) in a data plane of a network element and a determination made that the packet is a multicast packet. The packet is forwarded to all egress FDUs having at least one port associated with at least one receiver of the multicast packet. A lookup is performed by each egress FDU in a synchronized local port state database to find a port for each receiver that is in an UP state. The packet is forwarded out the port to a receiver when the port is in the UP state and dropped when the port is in the DOWN state.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit of U.S. Provisional PatentApplication No. 61/178,011, filed on May 13, 2009, which is incorporatedherein by reference in its entirety. This application also relates tothe application titled “Method And Apparatus For Maintaining Port StateTables In A Forwarding Plane Of A Network Element”, Attorney DocketNumber AVA10-17, filed on the same date as the present application. Theteachings and disclosure of the above-identified applications are eachincorporated by reference herein in their entirety.

BACKGROUND

Data communication networks may include various computers, servers,nodes, routers, switches, hubs, proxies, and other devices coupled toand configured to pass data to one another. These devices are referredto herein as “network elements,” and may provide a variety of networkresources on a network. Data is communicated through data communicationnetworks by passing protocol data units (such as packets, cells, frames,or segments) between the network elements over communication links onthe network. A particular protocol data unit may be handled by multiplenetwork elements and cross multiple communication links as it travelsbetween its source and its destination over the network. Hosts such ascomputers, telephones, cellular telephones, Personal Digital Assistants,and other types of consumer electronics connect to and transmit/receivedata over the communication network and, hence, are users of thecommunication services offered by the communication network.

Network elements are typically implemented to have a control plane thatcontrols operation of the network element and a data plane that handlestraffic flowing through the network. The data plane typically will havea collection of line cards having ports that connect to links on thenetwork. Data is received at a particular port, switched within the dataplane, and output at one or more other ports onto other links on thenetwork. To enable the data to be handled quickly, the data plane istypically implemented in hardware so that all of the decisions as to howto handle the data are performed using hardware lookups, etc.

Ports can fail for many reasons, including line card failure, failure ofthe link connected to the port (e.g. line cut), far-end line cardfailure, etc. The terms multi-link trunk (MLT), Link Aggregation Group(LAG) and logical ports are synonymous and these terms are usedinterchangeably Likewise, the internal forwarding datapath within thenetwork element may fail which may cause a port or set of ports toappear to have failed, or there may be some other failures along thelogical/virtual connection to the port's external peer endpoint. Thereare numerous reasons why a port may fail.

In the event a port fails, traffic destined to the port should bediverted to flow out an alternate port to enable connectivity to berestored through the network. To minimize impact on the traffic beinghandled by the network element, e.g. to minimize down-time and packetloss, the quicker the rerouting of traffic can occur the better.Preferably, it would be advantageous to enable the traffic to fail overto an alternate port in under ten milliseconds (ms). Preferably, in thecase of LAG or MLT, the traffic should be spread across the remainingports rather than all moved from the failing port to a particulardesignated alternate port to prevent the designated alternate port frombeing overloaded with traffic.

SUMMARY

Conventional mechanisms such as those explained above suffer from avariety of deficiencies. One such deficiency is the amount of time takenby conventional network elements to recover from a failure. The longerthe amount of time taken to recover the longer the delay in performance(including more dropped packets) as well as the loading of other portsto overcome for the failed port.

Embodiments of the invention significantly overcome such deficienciesand provide mechanisms and techniques that provide a fast re-route of amulticast packet within a network element to an available portassociated with a multi-link trunk. In a particular embodiment themethod includes receiving a packet by the FDU in a data plane of anetwork element and determining that the packet is a multicast packet.The method further includes forwarding the packet to all egress FDUshaving at least one port associated with at least one receiver of themulticast packet. Additionally, the method includes performing a lookupby each egress FDU in a synchronized local port state database to find aport for each receiver that is in an UP state.

Other embodiments include a computer readable medium having computerreadable code thereon for providing a fast re-route of a multicastpacket within a network element to an available port associated with amulti-link trunk. The computer readable medium includes instructions forreceiving a packet by the FDU in a data plane of a network element anddetermining that the packet is a multicast packet. The computer readablemedium further includes instructions for forwarding the packet to allegress FDUs having at least one port associated with at least onereceiver of the multicast packet. Additionally, the computer readablemedium includes instructions for performing a lookup by each egress FDUin a synchronized local port state database to find a port for eachreceiver that is in an UP state.

Still other embodiments include a computerized device, configured toprocess all the method operations disclosed herein as embodiments of theinvention. In such embodiments, the computerized device includes amemory system, a processor, communications interface in aninterconnection mechanism connecting these components. The memory systemis encoded with a process that provides a fast re-route of a multicastpacket within a network element to an available port associated with amulti-link trunk as explained herein that when performed (e.g. whenexecuting) on the processor, operates as explained herein within thecomputerized device to perform all of the method embodiments andoperations explained herein as embodiments of the invention. Thus anycomputerized device that performs or is programmed to perform upprocessing explained herein is an embodiment of the invention.

Other arrangements of embodiments of the invention that are disclosedherein include software programs to perform the method embodiment stepsand operations summarized above and disclosed in detail below. Moreparticularly, a computer program product is one embodiment that has acomputer-readable medium including computer program logic encodedthereon that when performed in a computerized device provides associatedoperations providing a fast re-route of a multicast packet within anetwork element to an available port associated with a multi-link trunkas explained herein. The computer program logic, when executed on atleast one processor with a computing system, causes the processor toperform the operations (e.g., the methods) indicated herein asembodiments of the invention. Such arrangements of the invention aretypically provided as software, code and/or other data structuresarranged or encoded on a computer readable medium such as an opticalmedium (e.g., CD-ROM), floppy or hard disk or other a medium such asfirmware or microcode in one or more ROM or RAM or PROM chips or as anApplication Specific Integrated Circuit (ASIC) or as downloadablesoftware images in one or more modules, shared libraries, etc. Thesoftware or firmware or other such configurations can be installed ontoa computerized device to cause one or more processors in thecomputerized device to perform the techniques explained herein asembodiments of the invention. Software processes that operate in acollection of computerized devices, such as in a group of datacommunications devices or other entities can also provide the system ofthe invention. The system of the invention can be distributed betweenmany software processes on several data communications devices, or allprocesses could run on a small set of dedicated computers, or on onecomputer alone.

It is to be understood that the embodiments of the invention can beembodied strictly as a software program, as software and hardware, or ashardware and/or circuitry alone, such as within a data communicationsdevice. The features of the invention, as explained herein, may beemployed in data communications devices and/or software systems for suchdevices such as those manufactured by Avaya, Inc. of Lincroft, N.J.

Note that each of the different features, techniques, configurations,etc. discussed in this disclosure can be executed independently or incombination. Accordingly, the present invention can be embodied andviewed in many different ways.

Also, note that this summary section herein does not specify everyembodiment and/or incrementally novel aspect of the present disclosureor claimed invention. Instead, this summary only provides a preliminarydiscussion of different embodiments and corresponding points of noveltyover conventional techniques. For additional details, elements, and/orpossible perspectives (permutations) of the invention, the reader isdirected to the Detailed Description section and corresponding figuresof the present disclosure as further discussed below.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing will be apparent from the following more particulardescription of preferred embodiments of the invention, as illustrated inthe accompanying drawings in which like reference characters refer tothe same parts throughout the different views. The drawings are notnecessarily to scale, emphasis instead being placed upon illustratingthe principles of the invention.

FIG. 1 is a functional block diagram showing a first example oforganizing a cluster of nodes;

FIG. 2 is a functional block diagram showing another example oforganizing a cluster of nodes;

FIG. 3 is a functional block diagram showing another example oforganizing a cluster of nodes; FIG. 4 is a functional block diagramshowing another example of organizing a cluster of nodes;

FIG. 5 is a functional block diagram showing another example oforganizing a cluster of nodes;

FIG. 6 is a functional block diagram showing another example oforganizing a cluster of nodes;

FIG. 7 is a functional block diagram of an example communicationnetwork;

FIG. 8 is a functional block diagram of an example network element;

FIG. 9 is a functional block diagram of an example line card that may beused in a network element such as the network element of FIG. 8;

FIG. 10 is a functional block diagram of an example port database thatmay be used in a line card such as the line card of FIG. 9;

FIG. 11 is a functional block diagram of an example cluster of networkelements showing the flow of port state information between thecomponents of the datapath within the node cluster;

FIG. 12 illustrates an example computer system architecture for anetwork element that operates in accordance with embodiments of theinvention;

FIG. 13 provides a fast re-route of a multicast packet within a networkelement to an available port associated with a multi-link trunk;

FIG. 14 provides a fast re-route of a multicast packet within a networkelement to an available MLT group for a local FDU associated with amulti-link trunk;

FIG. 15 provides a fast re-route of a multicast packet within a networkelement to an available MLT group for a remote FDU associated with amulti-link trunk.

DETAILED DESCRIPTION

Over time, the manner in which network elements handle data has evolved.For example, two or more physical links may extend between a group ofnetwork elements and be used collectively as a MLT or LAG. FIG. 1 showsan example of two network elements (network element 1 and networkelement 2) connected by multiple links 24 a-d that have been grouped toform a multi-link trunk 22. In particularly, each of the links 24 a-d inthe MLT 22 may be used by either of the network elements to forward datato the other. Thus, if network element 1 has data (e.g. a frame/packet)to send to network element 2, network element 1 may select one of thelinks 24 a-24 d from the MLT 22 and transmit the packet over that linkto network element 2.

FIG. 2 shows another example way in which network elements may beinterconnected. Specifically, in this example network element 1 andnetwork element 2 are interconnected by an inter-switch trunk (IST) 26which may be a single link or itself may be a multi-link trunk. When thelinks of a multilink trunk are physically connected to two differentnetwork elements, the MLT is called a Split Multi-Link Trunk (SMLT).Network elements 1 and 2 may each have one or more links that connectsto network element 3, which may be grouped together to form a SMLT 23.Thus, if network element 1 has data (e.g. a frame/packet) to send tonetwork element 3, network element 1 may either select one of the SMLTlinks connected to it or may transmit the packet on one of the linksassociated with the Inter-Switch Trunk 26 to enable the network element2 to forward the data on one of its links associated with the SMLT tonetwork element 3.

FIG. 3 shows another example in which network element 1 does not haveany links connected to the SMLT 23, but is connected by ISTs 25 and 26to two other network elements (network element 2 and network element 3)that do have ports connected to the links associated with the SMLT. Inthis scenario, if network element 1 has data to send on the SMLT, itwill select one of the IST links (note that each IST link may itself bea SMLT) and forward the data on to either network element 2 or 3. TheISTs may be physical and extend directly between two network elements ormay be logical and extend on tunnels through one or more intermediatenetwork elements.

FIG. 4 shows another example in which network element 1 alsoparticipates in the SMLT 23. In this instance, if network element 1 hasdata to send, it may forward the data on one of its links associatedwith the SMLT 23 or may forward the data on one of the links associatedwith one of the ISTs 25 or 26 to enable the data to be forwarded on theSMLT 23.

FIGS. 5 and 6 show another way of interconnecting network elements in asquare SMLT arrangement. In the arrangement shown in FIG. 5, fournetwork elements are interconnected via ISTs 25-28 in a squarearrangement, and in FIG. 6 the four network elements are interconnectedvia ISTs 25-28 in a meshed arrangement. The ISTs may be physical andextend directly between two network elements or may be logical andextend on tunnels through one or more intermediate network elements.

Although several examples of ways in which network devices may beinterconnected have been shown, there are other ways to interconnect acluster of network elements as well and this example set ofinterconnection architectures is not intended to be exhaustive. Thus,these examples were merely intended to provide a representative exampleof a few ways of interconnecting network elements. A group of networkelements will be referred to herein as a cluster.

FIG. 7 shows an example communication network 10 in which subscribers 12connect to an edge switch 14. The edge switch 14 connects to coreswitches 16 which forward data through the network on links 18. Each ofthese switches may be a physical rouswitchter or may be multiple devicesconnected together to operate as a cluster. Each of the links 18 may bea MLT or, where the router/switch is implemented as multiple physicaldevices, may be a SMLT. From a network routing standpoint, there may bemultiple ways for a packet to traverse the network. For example, in FIG.7 the edge switch A may be able to transmit a packet to edge switch Bthrough core switches C and D or, alternatively, may be able to transmitthe packet through core switches E and F. A network layer routingprotocol may be used to determine the path to be used for transmissionof the packet.

As noted above, depending on the manner in which the network elementsare interconnected, there may be many ways for the network element toforward a frame/packet to enable the frame/packet to reach itsdestination. As used herein, the term “cluster” is used to refer to oneor more nodes providing node-level resiliency at the network level.Thus, in FIG. 1, network element 1 would be a cluster; in FIG. 2 networkelements 1 and 2 would be a cluster, and in FIGS. 3 and 4 networkelements 1, 2, and 3 would be a cluster and in FIGS. 5 and 6 networkelements 1-4 would be a cluster. As noted above there are other ways oforganizing nodes within a cluster.

Logical connections between the cluster nodes are referred to herein asInter-Switch Trunks (ISTs). ISTs may be physical links that extend fromone network element to a neighboring network element in the cluster, ormay be logical links that tunnel through one or more intermediatenetwork elements within the cluster. The node that receives a packetfrom a non-IST port will be referred to as a local node. All other nodeswithin the cluster are referred to as remote nodes with respect to thereceived packet.

Two or more links may be grouped to form a Multi-Link Trunk (MLT). EachMLT will be assigned a MLT group ID (MLT-ID), which is a global valuewithin the cluster and unique across the cluster nodes. An MLT with allits port members only on one node is referred to as a normal MLT group.An MLT group where its port members are on two or more nodes is referredto as a Split MLT or SMLT group.

When a logical port is implemented as a MLT or SMLT, there are actuallymultiple physical ports that are capable of forwarding a packet to itsnext hop on the network. Accordingly, if one of the ports of a MLT/SMLTfails, it would be advantageous to cause the packet to be forwarded onone of the remaining ports so that the packet can traverse the networkrather than being dropped. Likewise, rather than designate a primary andbackup port for each port in the MLT/SMLT, it would be advantageous toload share the packets across the remaining ports of the MLT/SMLT sothat the packets may be distributed across the remaining ports that areUP. According to an embodiment, this process is implemented in hardwareso that the fastpath (dataplane) can automatically accommodateindividual and multiple port failures and automatically redirect packettraffic across the remaining ports in an equitable manner.

FIG. 8 shows a functional block diagram of an example network element 20which may be used as any of the network elements shown in any of FIGS.1-6. In the example shown in FIG. 8, the network element includes acontrol plane 31 and a data plane 32. The control plane has one or moreCPUs 34 and generally run control processes such as routing processes,management processes, etc. The control plane programs the data plane toinstruct the data plane how to forward data on the network.

The data plane 32 may be constructed using many different architecturesand the example shown in FIG. 8 is only one example of one sucharchitecture. In the example shown in FIG. 8, the data plane includes aplurality of line cards 36 each of which implements multiple physicalports which connect to links in the network. The line cards in thisembodiment are interconnected by a switch fabric 40, although in otherembodiments the line cards may be directly interconnected and performswitching functions in a distributed manner.

As shown in FIG. 9, each line card 36 includes a plurality of ports 38which physically connect to the links on the network. The line card alsoincludes one or more functional units 42 that process packets receivedfrom the attached ports. As used herein, the functional unit thatprocesses packets from attached ports in both ingress and egressdirections, and makes forwarding decisions, is referred to as aForwarding Datapath Unit or FDU 42. The line card may also include a CPU44 that interacts with the control plane to enable the control plane toprogram instructions into the FDU 42 and optionally other components onthe line card so that the FDU 42 will handle data appropriately on thenetwork. The CPU 44 also periodically checks the status of the FDU 42and other components of the line card to detect when a failure occurs.

Each FDU 42 maintains a port state database 46. This port state database46 maintains the physical link states and connection states for itslocal as well as all remote ports. As shown in FIG. 10, the port statedatabase 46 includes two tables—a local port data table 48 and a remoteport data table 50. The local port data table 48 maintains the portstates belonging to the local node, and the remote port data table 50maintains the port states belonging to the remote nodes within thecluster. In the case of MLT groups, the FDU keeps the MLT port memberson the local node in the local table, and the port members on all otherremote nodes in the remote table.

When the FDU receives a packet, it is required to find a port within theMLT (or SMLT) that is UP to forward the packet on toward its destinationon the network. As noted above, where all the ports on the MLT arelocal, the FDU will need to determine which of the local ports is in theUP state so that it does not attempt to forward the packet over a portthat is DOWN. Likewise, where the ports associated with the MLT are notall local (e.g. SMLT), the FDU will need to select a port on a separatephysical network element that is associated with the SMLT and has an UPstate.

According to an embodiment, each FDU maintains a port state database 46that it uses to maintain the state of each port within the cluster. Thedatabase maintains the physical link states and connection states forits local as well as all remote ports. The database includes twotables—namely a local table 48 and remote table 50. The local tablemaintains the port states belonging to all FDUs on the local node, andthe remote table maintains the port states of all ports on all remotenodes within the cluster. In the case of MLT groups, the FDU keeps thestate of the MLT port members that are on the local node within thelocal table, and keeps the state of the MLT port members that are on allother remote nodes in the remote table. The local table also maintainsthe state of all IST ports. When a packet is received, the FDU will usethe port state database to determine a port for the packet that is UPand forward the packet to that port to be output on the network.

Since the port state database 46 is used by the FDU 42 to makeforwarding decisions, it is important to keep the port state tableupdated, so that it contains current information about the state of eachof the ports. Since each line card has one or more FDU, each FDU (ineach line card) is required to synchronize with the FDUs in all otherline cards within the local network element as well as with all otherFDUs in other network elements within the cluster.

In one embodiment, each line card maintains heartbeat timers. Each timea particular one of the heartbeat timers expires, a heartbeat engine 54generates a heartbeat packet and sends the heartbeat packet to the localFDU 42 on that line card. The heartbeat packet carries the localphysical link status of all ports on the line card to inform the localFDU of the state of the ports on that line card. The packet indicatesthe Global Port ID (GPID) and a network element ID. The FDU uses thisstate to update its local port state database. The FDU will also forwardthe packet to all of the other FDUs within the local node, as well as toall other FDUs on other nodes within the cluster. Each FDU uses the portstate carried by the packet to update its port state database.

In addition to maintaining a timer associated with collecting/reportingits own port state information, each line card/FDU will also maintain aset of timers associated with all other FDUs within the local node andall other FDUs on other nodes within the cluster. Each FDU expects toreceive periodic heartbeat packets from every other local and remoteFDU. Thus, a local reception timer is maintained per FDU (for each FDUon the local network element as well as for each FDU on each remotenetwork element within the cluster). A FDU failure (local or remote) isdetected if the corresponding reception timer expires. Where a heartbeatpacket is not received before expiration of the reception timer, eachport associated with the FDU will be set to DOWN so that packets are notsent to ports associated with that FDU until it is restored.

The heartbeat packets allow the FDUs to convey state information to eachother and allow each FDU to know the state of all ports in thedataplane. As described below, this allows the dataplane toautomatically adjust to port failures so that data may be redirected toports that are UP and away from ports that are Down. All this happenswithout intervention from the control plane and, hence, the controlplane is not notified of a failure of a particular port/line card. Toenable the control plane to learn of dataplane failures, themanagement/control processor 44 periodically injects and extractsheartbeat packets into and out of its local FDU 42. Each injectedheartbeat packet completely loops through the target FDU and associatedports and then is extracted back to the processor. The managementheartbeat packet traverses all functional blocks in both ingress andegress datapaths. Each time the control processor injects a managementheartbeat packet of this nature, it kicks off its correspondingreception timer. The control processor detects a failure of the linecard if the reception timer expires. The processor uses this informationto set a system alarm which will be conveyed to the control plane 30.The control plane may thus learn about a data plane failure. However,since the dataplane has a self-healing mechanism to accommodate portfailures and redirect traffic accordingly, the control plane is notrequired to be involved in redirecting traffic and, hence, notificationof the control plane of the failure is not critical to restoration oftraffic through the network element.

Heartbeat packets are also used by each FDU to determine the state ofits ports. In one embodiment, each FDU maintains a pair of timers perattached port that is configured in a logical/virtual connection. One ofthe timers is used to generate heartbeat packets to be sent over theconnection. The other timer (reception timer) is used to detectconnection failure. This timer expires if the heartbeat packet from theother endpoint of the connection is not received in time. The FDUupdates its port state table with the arrival of heartbeat packets andreception timer expirations.

Each FDU on each line card maintains its own port state table 46. Thistable maintains the physical link states and connection states for itslocal ports as well as all remote ports of all FDUs in the cluster. TheFDU uses the received heartbeat packets and timer expiration messages(due to connection time-out or remote FDU failure) to update the table.The table is partitioned into two segments: port states belonging tolocal node and port states belonging to the remote nodes. The port statetable also maintains MLT and SMLT group information. The port statetable is used by the forwarding logic to perform fast reroute asexplained in greater detail below.

FIG. 11 shows dissemination of the port state packet within a cluster offour nodes, in which the dark arrows represents the flow of the portstate packet to all FDUs in the cluster. As shown in this figure, packet1 will be generated containing the state of each port associated with aparticular FDU. This packet is shown with reference numeral 1 in the topleft line card of the top left network element. This packet will bepassed to the FDU so that the FDU can update its port state database toreflect the current status of its ports. The packet will then be passedto each of the other FDUs within the local node (arrows 2). In oneembodiment this may be implemented by causing the packet to be broadcastby the switch fabric to all other FDUs within the local node. The packetwill also be forwarded to other nodes within the cluster (arrows 3) sothat the state of the port may be distributed (arrows 4) to each FDUassociated with each node of the cluster. Whenever a FDU receives apacket containing port state information, it will use the information toupdate its own port state database. This enables the port state databaseof all FDUs in the cluster to be synchronized.

There may be several network elements within a cluster, multiple FDUswithin a network element, and multiple ports supported by each FDU. Toenable each node to keep track of which FDUs have provided statepackets, and to correlate particular port state packets with particularFDUs, a numbering scheme may be implemented. Preferably the numberingscheme is implemented to be cluster wide unique so that each FDU withinthe cluster may be uniquely identified. In one embodiment, the portstate packet carries information about each of its ports. The packetwill specify the source node ID and the Global Port ID (GPID). TheGlobal Port ID is the globally unique identifier (globally unique withina node) that enables the port to be uniquely identified within the portdatabase.

The previous messages described how the FDUs exchanged messages toenable port state to be synchronized between ports in the datapath. Inoperation, this port state information will enable the FDUs to select anavailable port for a particular data packet with confidence that theselected port is UP. As ports go Down, the FDUs in the cluster will stopselecting those ports and will instead select alternate ports within theMLT/SMLT associated with the down port to be used to handle the packet.Accordingly, the datapath is able to automatically accommodate portfailures, line card failures, etc., to re-route packets to availablealternate ports without involvement of the control plane. Hence,rerouting of packets may be done quickly within a network element andbetween clusters of network elements in under 10 ms.

When a FDU receives a data packet it will read the port ID, MLT-ID, andhash function, and pass the these values to the port state database. Theport state database will search first for a local port that isassociated with the MLT-ID and which is UP, and then send for a remoteport that is associated with the MLT and is UP. In one embodiment, theport state table is designed to offload port selection processing fromthe FDU. In this embodiment, the FDU passes a request for a port to theport state table and the port state table intelligently implements therequest to return a port that is guaranteed to be UP, which is part ofthe MLT/SMLT, and with preference to use of local ports over remoteports. Additionally, where the port is on a remote node, the port statetable will check to find an IST port that is UP over which the FDU canforward the packet to the remote node, so that the remote node canforward the packet on the remote port. Thus, the port state table notonly determines whether there is an available remote port, but alsowhether there is an available IST port that the FDU can use to pass thepacket to the remote node for forwarding over the remote port.

FIG. 10 shows an embodiment that may be used to offload port selectionfrom the FDU. In the embodiment shown in FIG. 10, the port databaseincludes database control logic 56 configured to receive requests forport selection from the FDU and return an available port if one exists.The port database may need to make multiple database accessoperations/calls to the local and remote port tables to determine whichport should be returned in response to the port selection request. Thedatabase control logic controls execution of this process, optionallyunder the control of the FDU, to automate the process of selecting aport.

The database control logic may be implemented in hardware and performdata accesses in the local and remote tables which are also implementedin hardware. As ports fail, the status (UP/Down) of the port will bereflected in the port state table. The FDU will perform a port accessoperation via the database control logic into the port state tablewhenever the FDB lookup returns a destination that is part of anMLT/SMLT group. Accordingly, rather than having the FDU determine theport from the FDB (which is updated by the control pane and, hence, isrelatively slow), the FDU uses the FDB to determine the receiver andthen separately determines the port to be used to reach the receiverusing the hardware-implemented port database. This enables the port tobe determined dynamically based on the current state of the ports andthe MLT group ID, so that an available port from the group of associated(MLT) ports can be selected for the packet. This enables the networkdevice to accommodate multiple failures (multiple port/link failures)and quickly adapt by re-routing the packets away from ports that aredown to ports that are guaranteed to be UP within the MLT. Since portfailure notifications span multiple network devices within the cluster,the FDU may forward a packet to another port on a different node so thatport selection within and between network devices may be implemented ina similar manner. This enables not only port selection from within a MLThaving only local ports, but enables port selection to be implementedbetween ports implemented on separate nodes within a cluster so thatport selection within a SMLT may be dynamically implemented.Additionally, since the nodes are able to select from between all otherports associated with the MLT/SMLT, traffic from the port that is downmay be spread/distributed across the remaining ports of the MLT/SMLTrather than being shifted entirely to a particular alternate port.Likewise, multiple port failures and combinations of port failures maybe accommodated without disrupting packet forwarding through the nodecluster, since whenever a packet is received the port state databasewill be able to look for any available port that is UP and return thevalue of any available port for use by the FDUs.

FIG. 12 is a block diagram illustrating example architecture of acomputer system (FDU) 110 that executes, runs, interprets, operates orotherwise performs a fast reroute operating application 140-1 and fastreroute operating process 140-2 suitable for use in explaining exampleconfigurations disclosed herein. As shown in this example, the computersystem 110 includes an interconnection mechanism 111 such as a data busor other circuitry that couples a memory system 112, a processor 113, aninput/output interface 114, and a communications interface 115. Thecommunications interface 115 enables the computer system 110 tocommunicate with other devices (i.e., other computers) on a network (notshown).

The memory system 112 is any type of computer readable medium, and inthis example, is encoded with a fast reroute operating application 140-1as explained herein. The fast reroute operating application 140-1 may beembodied as software code such as data and/or logic instructions (e.g.,code stored in the memory or on another computer readable medium such asa removable disk) that supports processing functionality according todifferent embodiments described herein. During operation of the computersystem 110, the processor 113 accesses the memory system 112 via theinterconnect 111 in order to launch, run, execute, interpret orotherwise perform the logic instructions of a fast reroute operatingapplication 140-1. Execution of a fast reroute operating application140-1 in this manner produces processing functionality in the fastreroute operating process 140-2. In other words, the fast rerouteoperating process 140-2 represents one or more portions or runtimeinstances of a fast reroute operating application 140-1 (or the entire afast reroute operating application 140-1) performing or executing withinor upon the processor 113 in the computerized device 110 at runtime.

It is noted that example configurations disclosed herein include thefast reroute operating application 140-1 itself (i.e., in the form ofun-executed or non-performing logic instructions and/or data). The fastreroute operating application 140-1 may be stored on a computer readablemedium (such as a floppy disk), hard disk, electronic, magnetic,optical, or other computer readable medium. A fast reroute operatingapplication 140-1 may also be stored in a memory system 112 such as infirmware, read only memory (ROM), or, as in this example, as executablecode in, for example, Random Access Memory (RAM). In addition to theseembodiments, it should also be noted that other embodiments hereininclude the execution of a fast reroute operating application 140-1 inthe processor 113 as the fast reroute operating process 140-2. Thoseskilled in the art will understand that the computer system 110 mayinclude other processes and/or software and hardware components, such asan operating system not shown in this example.

During operation, processor 113 of computer system 100 accesses memorysystem 112 via the interconnect 111 in order to launch, run, execute,interpret or otherwise perform the logic instructions of the fastreroute application 140-1. Execution of fast reroute application 140-1produces processing functionality in fast reroute process 140-2. Inother words, the fast reroute process 140-2 represents one or moreportions of the fast reroute application 140-1 (or the entireapplication) performing within or upon the processor 113 in the computersystem 100.

It should be noted that, in addition to the fast reroute process 140-2,embodiments herein include the fast reroute application 140-1 itself(i.e., the un-executed or non-performing logic instructions and/ordata). The fast reroute application 140-1 can be stored on a computerreadable medium such as a floppy disk, hard disk, or optical medium. Thefast reroute application 140-1 can also be stored in a memory typesystem such as in firmware, read only memory (ROM), or, as in thisexample, as executable code within the memory system 112 (e.g., withinRandom Access Memory or RAM).

In addition to these embodiments, it should also be noted that otherembodiments herein include the execution of fast reroute application140-1 in processor 113 as the fast reroute process 140-2. Those skilledin the art will understand that the computer system 100 can includeother processes and/or software and hardware components, such as anoperating system that controls allocation and use of hardware resourcesassociated with the computer system 100.

Referring now to FIG. 13 a flow diagram of a particular embodiment of amethod 200 for performing a fast re-route of a multicast packet within anetwork element to an available port associated with a multi-link trunkis shown. Method 200 starts with processing block 202 which disclosesreceiving a packet by said FDU in a data plane of a network element.

Processing block 204 states determining that the packet is a multicastpacket. Multicast (and broadcast) operate such that the packet will needto be replicated by the node so that multiple copies of the packet maybe sent out to multiple different receivers. When a FDU receives amulticast packet, it will get the multicast group ID from the FDB. Withmulticast, since there is a possibility that the network element willneed to forward the packet to multiple receivers, the ingress FDU doesnot perform port selection but rather simply forwards the packet to theswitch fabric.

Processing block 206 recites forwarding the packet to all egress FDUshaving at least one port associated with at least one receiver of themulticast packet. The switch fabric uses the multicast group ID to makecopies of the packet to all FDUs that support at least one portassociated with at least one receiver on the multicast. In an embodimentwhere the line cards are directly connected, the FDU will forward a copyof the packet to each other line card supporting at least one portassociated with at least one receiver. In this example, as shown inprocessing block 208, a determination is made that the receiver port isa stand alone port. If a particular receiver is on a stand-alone port,then the switch fabric will forward a copy of the packet to the FDUsupporting that stand-alone port.

Processing block 210 discloses performing a lookup by each egress FDU ina synchronized local port state database to find a port for eachreceiver that is in an UP state. The FDU will then access the databasecontrol logic to determine if the standalone port is UP or down.Processing block 212 states forwarding said packet out said port to areceiver when said port is in the UP state and dropping said packet whensaid port is in the DOWN state.

Referring now to FIG. 14 a flow diagram of a particular embodiment of amethod 250 for performing a fast re-route of a multicast packet within anetwork element to an available MLT group for a local FDU associatedwith a multi-link trunk is shown. Method 250 starts with processingblock 252 which discloses receiving a packet by said FDU in a data planeof a network element.

Processing block 254 states determining that the packet is a multicastpacket. When a FDU receives a multicast packet, it will get themulticast group ID from the FDB. With multicast, since there is apossibility that the network element will need to forward the packet tomultiple receivers, the ingress FDU does not perform port selection butrather simply forwards the packet to the switch fabric.

Processing block 256 recites forwarding the packet to all egress FDUshaving at least one port associated with at least one receiver of themulticast packet. The switch fabric uses the multicast group ID to makecopies of the packet to all FDUs that support at least one portassociated with at least one receiver on the multicast. In an embodimentwhere the line cards are directly connected, the FDU will forward a copyof the packet to each other line card supporting at least one portassociated with at least one receiver. In this example, as shown inprocessing block 258, a determination is made that the receiver port isa stand alone port. In this example, as shown in processing block 258, adetermination is made that the receiver port is an MLT group for a localFDU.

Processing block 260 discloses performing a lookup by each egress FDU ina synchronized local port state database to find a port for eachreceiver that is in an UP state. Where a receiver is able to be reachedvia a MLT group, the switch fabric will forward a copy of the packet toeach FDU associated with a port that is part of the MLT group. Where thereceiver is part of a SMLT group, the switch fabric will also replicatea copy of the packet to each FDU associated with an IST to the remotenodes supporting the remote ports. Where a given FDU supports more thanone port associated with a particular receiver, or supports more thanone port associated with more than one particular receiver, only onecopy of the packet will be forwarded to that FDU. Only one copy will besent over the IST, and the remote node will make copies for each portmember. As further shown in processing block 262, a MLT Identifier (ID)and hash value are used to perform said lookup in said local port statedatabase. Each local FDU that receives a copy of the replicated packetwill access the port table by passing a hash of selected fields of thepacket and the MLT-ID to the port database. The MLT ID and hash valueare used by the port database to look for a local port member that isUP. Each local FDU will perform this same process using the same MLT-ID,hash value, and synchronized port database. Hence, each port databasewill return the same local port ID (if a local port ID is available i.e.UP). Any FDU that supports the selected port will transmit the packetout the selected port. Any FDU that does not support the selected portwill drop the packet. If no local port is available, all local FDUs willdrop the packet. Stated differently, since the process of selecting aport is performed at egress from the switch fabric, where the MLT spansmultiple line cards and nodes, the process will be performed in parallelby each of the FDUs that service one or more ports associated with theMLT/SMLT to the receiver. This process also is performed by the FDU/setof FDUs for each receiver associated with the broadcast or multicast.

Each FDU (associated with a particular receiver) will create a hashfunction based on selected fields of the packet header and will pass thehash function and MLT-ID to its local port database. Since each FDU hasa synchronized port database and passes the same request (hash functionand MLT-ID) to its local port database, each port database will return aconsistent output port to be used to forward the packet on toward thereceiver.

Processing block 264 states forwarding said packet out said port that isin an UP state and dropping said packet by all remote FDUs. When a FDUreceives the response from its local port database, it will determinewhether it supports the selected port or whether the selected port issupported by another FDU. All FDUs other than the FDU that supports theselected port will then drop the packet so that a single copy of thepacket may be forwarded to the receiver.

Referring now to FIG. 15 a flow diagram showing a particular embodimentof a method 300 for providing a fast re-route of a multicast packetwithin a network element to an available MLT group for a remote FDUassociated with a multi-link trunk is presented. Method 300 begins withprocessing block 302 which recites receiving a packet by said FDU in adata plane of a network element.

Processing block 304 discloses determining that the packet is amulticast packet. When a FDU receives a multicast packet, it will getthe multicast group ID from the FDB. With multicast, since there is apossibility that the network element will need to forward the packet tomultiple receivers, the ingress FDU does not perform port selection butrather simply forwards the packet to the switch fabric.

Processing block 306 states forwarding the packet to all egress FDUshaving at least one port associated with at least one receiver of themulticast packet. The switch fabric uses the multicast group ID to makecopies of the packet to all FDUs that support at least one portassociated with at least one receiver on the multicast. In an embodimentwhere the line cards are directly connected, the FDU will forward a copyof the packet to each other line card supporting at least one portassociated with at least one receiver. In this example, as shown inprocessing block 308, a determination is made that the receiver port isan MLT group for a remote FDU. In an SMLT context, if one or more of theports associated with a particular receiver is on a remote node, a copyof the packet will be forwarded to the remote node on the IST. When theappropriate FDU on the remote node receives the packet, it will checkits port database to determine whether it is required to forward a copyof the packet on one of the packets associated with the SMLT.

Processing block 310 discloses performing a lookup by each egress FDU ina synchronized local port state database to find a port for eachreceiver that is in an UP state. As further shown in processing block312, a MLT Identifier (ID) and hash value are used to perform saidlookup in said remote port state database. To do this, each remote FDUwill first check to see whether any of the ports associated with theSMLT on the originating node are UP.

Processing block 314 states checking remote port tables to determinewhether any MLT ports on the originating network element are in the UPstate. Processing block 316 recites when one MLT port on the originatingnetwork element are in the UP state then dropping said packet by saidremote FDUs. Note, in this regard, that the packet will preferentiallybe transmitted out a port on the local node that first received thepacket. Accordingly, if one or more of the SMLT ports on the local nodethat first received the packet is UP, the remote nodes may safelydiscard the packet.

Processing block 318 discloses when all MLT ports on the originatingnode are down then selecting one remote port to transmit said packet anddropping said packet by all other remote ports. If all of the SMLT portson the local node that first received the packet are Down, then eachFDUs on the remote nodes will query their port database to select aremote SMLT port for transmission of the packet. The remote FDUs havesynchronized port databases and will use the same hash value/MLT-ID toquery their port databases, so that each remote FDU will receive thesame port ID from its port database. The one FDU that is responsible forthe selected port will then output the packet so that only one copy ofthe packet will be sent to the receiver regardless of the number ofavailable links of the SMLT.

By way of the above described methods and apparatus, a fast reroute of amulticast packet within a network element to an available port within amultilink trunk is performed.

The device(s) or computer systems that integrate with the processor(s)may include, for example, a personal computer(s), workstation(s) (e.g.,Sun, HP), personal digital assistant(s) (PDA(s)), handheld device(s)such as cellular telephone(s), laptop(s), handheld computer(s), oranother device(s) capable of being integrated with a processor(s) thatmay operate as provided herein. Accordingly, the devices provided hereinare not exhaustive and are provided for illustration and not limitation.

References to “a microprocessor” and “a processor”, or “themicroprocessor” and “the processor,” may be understood to include one ormore microprocessors that may communicate in a stand-alone and/or adistributed environment(s), and may thus be configured to communicatevia wired or wireless communications with other processors, where suchone or more processor may be configured to operate on one or moreprocessor-controlled devices that may be similar or different devices.Use of such “microprocessor” or “processor” terminology may thus also beunderstood to include a central processing unit, an arithmetic logicunit, an application-specific integrated circuit (IC), and/or a taskengine, with such examples provided for illustration and not limitation.

Furthermore, references to memory, unless otherwise specified, mayinclude one or more processor-readable and accessible memory elementsand/or components that may be internal to the processor-controlleddevice, external to the processor-controlled device, and/or may beaccessed via a wired or wireless network using a variety ofcommunications protocols, and unless otherwise specified, may bearranged to include a combination of external and internal memorydevices, where such memory may be contiguous and/or partitioned based onthe application. Accordingly, references to a database may be understoodto include one or more memory associations, where such references mayinclude commercially available database products (e.g., SQL, Informix,Oracle) and also proprietary databases, and may also include otherstructures for associating memory such as links, queues, graphs, trees,with such structures provided for illustration and not limitation.

References to a network, unless provided otherwise, may include one ormore intranets and/or the Internet, as well as a virtual network.References herein to microprocessor instructions ormicroprocessor-executable instructions, in accordance with the above,may be understood to include programmable hardware.

Unless otherwise stated, use of the word “substantially” may beconstrued to include a precise relationship, condition, arrangement,orientation, and/or other characteristic, and deviations thereof asunderstood by one of ordinary skill in the art, to the extent that suchdeviations do not materially affect the disclosed methods and systems.

Throughout the entirety of the present disclosure, use of the articles“a” or “an” to modify a noun may be understood to be used forconvenience and to include one, or more than one of the modified noun,unless otherwise specifically stated.

Elements, components, modules, and/or parts thereof that are describedand/or otherwise portrayed through the figures to communicate with, beassociated with, and/or be based on, something else, may be understoodto so communicate, be associated with, and or be based on in a directand/or indirect manner, unless otherwise stipulated herein.

Although the methods and systems have been described relative to aspecific embodiment thereof, they are not so limited. Obviously manymodifications and variations may become apparent in light of the aboveteachings. Many additional changes in the details, materials, andarrangement of parts, herein described and illustrated, may be made bythose skilled in the art.

Having described preferred embodiments of the invention it will nowbecome apparent to those of ordinary skill in the art that otherembodiments incorporating these concepts may be used. Additionally, thesoftware included as part of the invention may be embodied in a computerprogram product that includes a computer useable medium. For example,such a computer usable medium can include a readable memory device, suchas a hard drive device, a CD-ROM, a DVD-ROM, or a computer diskette,having computer readable program code segments stored thereon. Thecomputer readable medium can also include a communications link, eitheroptical, wired, or wireless, having program code segments carriedthereon as digital or analog signals. Accordingly, it is submitted thatthat the invention should not be limited to the described embodimentsbut rather should be limited only by the spirit and scope of theappended claims.

1. A computer-implemented method in which a Forwarding Data Unit (FDU)performs operations providing a fast re-route of a multicast packetwithin a network element to an available port associated with amulti-link trunk, the method comprising the steps of: receiving a packetby said FDU in a data plane of a network element; determining that thepacket is a multicast packet; forwarding the packet to all egress FDUshaving at least one port associated with at least one receiver of themulticast packet; and performing a lookup by each egress FDU in asynchronized local port state database to find a port for each receiverthat is in an UP state.
 2. The method of claim 1 wherein said forwardingthe packet to all egress FDUs having at least one port associated withat least one receiver of the multicast packet further comprises:determining if a receiver port is one of the group consisting of a standalone port, an MLT group for a local FDU, and a MLT group for a remoteFDU; and forwarding a copy to each MLT member.
 3. The method of claim 2wherein when said receiver port is a stand alone port then determiningwhether said standalone port is in an UP state or a DOWN state,forwarding said packet out said port to a receiver when said port is inthe UP state and dropping said packet when said port is in the DOWNstate.
 4. The method of claim 2 wherein when said receiver port is anMLT group for a local FDU then locating a local port member from thelocal port state table that is in an UP state.
 5. The method of claim 4further comprising forwarding said packet out said port that is in an UPstate and dropping said packet by all remote FDUs.
 6. The method ofclaim 4 wherein a MLT Identifier (ID) and hash value are used to performsaid lookup in said local port state database.
 7. The method of claim 2wherein when said receiver port is an MLT group for a remote FDU thenchecking remote port tables to determine whether any MLT ports onoriginating network element are in the UP state.
 8. The method of claim7 further comprising when one MLT port on the originating networkelement are in the UP state then dropping said packet by said remoteFDUs.
 9. The method of claim 7 further comprising when all MLT ports onthe originating node are down then selecting one remote port to transmitsaid packet and dropping said packet by all other remote ports.
 10. Themethod of claim 7 wherein a MLT Identifier (ID) and hash value are usedto perform said lookup in said remote port state database.
 11. AForwarding Data Unit (FDU) comprising: a memory; a processor; acommunications interface; an interconnection mechanism coupling thememory, the processor and the communications interface; and wherein thememory is encoded with an application providing a fast re-route of amulticast packet within a network element to an available portassociated with a multi-link trunk, that when performed on theprocessor, provides a process for processing information, the processcausing the FDU to perform the operations of: receiving a packet by saidFDU in a data plane of a network element; determining that the packet isa multicast packet; forwarding the packet to all egress FDUs having atleast one port associated with at least one receiver of the multicastpacket; and performing a lookup by each egress FDU in a synchronizedlocal port state database to find a port for each receiver that is in anUP state.
 12. The FDU of claim 11 wherein said forwarding the packet toall egress FDUs having at least one port associated with at least onereceiver of the multicast packet further comprises: determining if areceiver port is one of the group consisting of a stand alone port, anMLT group for a local FDU, and a MLT group for a remote FDU; andforwarding a copy to each MLT member.
 13. The FDU of claim 12 whereinwhen said receiver port is a stand alone port then determining whethersaid standalone port is in an UP state or a DOWN state, forwarding saidpacket out said port to a receiver when said port is in the UP state anddropping said packet when said port is in the DOWN state.
 14. The FDU ofclaim 12 wherein when said receiver port is an MLT group for a local FDUthen locating a local port member from the local port state table thatis in an UP state.
 15. The FDU of claim 14 further comprising forwardingsaid packet out said port that is in an UP state and dropping saidpacket by all remote FDUs.
 16. The FDU of claim 14 wherein a MLTIdentifier (ID) and hash value are used to perform said lookup in saidlocal port state database.
 17. The FDU of claim 12 wherein when saidreceiver port is an MLT group for a remote FDU then checking remote porttables to determine whether any MLT ports on originating network elementare in the UP state.
 18. The FDU of claim 17 further comprising when oneMLT port on the originating network element are in the UP state thendropping said packet by said remote FDUs.
 19. The FDU of claim 17further comprising when all MLT ports on the originating node are downthen selecting one remote port to transmit said packet and dropping saidpacket by all other remote ports.
 20. The FDU of claim 17 wherein a MLTIdentifier (ID) and hash value are used to perform said lookup in saidremote port state database.
 21. A computer readable storage mediumhaving computer readable code thereon for providing a fast re-route of amulticast packet within a network element to an available portassociated with a multi-link trunk, the medium including instructions inwhich a Forwarding Data Unit (FDU) performs operations comprising:receiving a packet by said FDU in a data plane of a network element;determining that the packet is a multicast packet; forwarding the packetto all egress FDUs having at least one port associated with at least onereceiver of the multicast packet, wherein said forwarding the packet toall egress FDUs having at least one port associated with at least onereceiver of the multicast packet further comprises determining if areceiver port is one of the group consisting of a stand alone port, anMLT group for a local FDU, and a MLT group for a remote FDU; whereinwhen said receiver port is a stand alone port then determining whethersaid standalone port is in an UP state or a DOWN state, and forwardingsaid packet out said port to a receiver when said port is in the UPstate and dropping said packet when said port is in the DOWN state;wherein when said receiver port is an MLT group for a local FDU thenlocating a local port member from the local port state table that is inan UP state, and forwarding said packet out said port to a receiver whensaid port is in the UP state and dropping said packet when a port in theUP state cannot be determined; and wherein when said receiver port is anMLT group for a remote FDU then checking remote port tables to determinewhether any MLT ports on originating network element are in the UPstate; and forwarding said packet out said port to a receiver when saidport is in the UP state and dropping said packet when a port in the UPstate cannot be determined.